After a breast conserving therapy, histology is often performed to check whether the excised tumor specimen is surrounded by a sufficient amount of normal tissue. If a positive margin is identified, then a second operation will be performed. Currently, the re-operation rate is 20-40%, which raises a need for intraoperative detection of residual cancers. The current intraoperative tools for margin assessment are time consuming and lack sufficient tumor specificity. Multiple emerging optical technologies improved the sensitivity and specificity but still suffered from a long procedure time. We propose a multi-modal spectroscopic photoacoustic/ultrasound imaging system to differentiate cancer and normal tissue in a resected breast tissue. Providing both chemical content and mechanical property information, this system, named MarginPAT, is able to measure map tissue components including blood, fat, fibrosis tissue, and breast lesion. Based on a compact Raman laser and a high-speed scanning chamber, the proposed MarginPAT system is able to scan 10 cm2 tissue area per min, which is faster than the frozen section analysis (gold standard for intraoperative assessment) by one order of magnitude. Such capacities allow 3 dimensional assessment of margin status in the entire excised tissue with >90% sensitivity and specificity.